[Bug] Preserve assistant reasoning_content on request forwarding (DeepSeek V4 thinking mode) - #3553
Open
Paramveersingh-S wants to merge 2 commits into
Open
Conversation
…Seek V4 thinking mode) Signed-off-by: Param <param15.veer.singh@gmail.com>
Paramveersingh-S
requested review from
AayushSaini101,
FAUST-BENCHOU,
WUKUNTAI-0211,
drivebyer,
ramkrishs,
shraderdm,
theohsiung and
wilsonwu
as code owners
September 7, 2026 14:42
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Paramveersingh-S
force-pushed
the
fix-reasoning-forwarding
branch
from
September 7, 2026 14:45
ce76757 to
30c6d64
Compare
Xunzhuo
requested changes
Sep 8, 2026
Xunzhuo
left a comment
Member
There was a problem hiding this comment.
Thanks for taking this on. The passthrough helpers are not wired into request processing on this head: the patch adds the helper and its isolated tests, while request_context.go only gains an unused encoding/json import and neither request-body pipeline calls extract or restore. The reported reasoning_content field is therefore still dropped, and the branch does not compile. Please capture it before the SDK parse, restore it after serialization, and add a processor-level round-trip regression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3547
When the semantic router forwards a multi-turn Chat Completions request to an upstream OpenAI-compatible endpoint (e.g. DeepSeek V4 in thinking mode), the
reasoning_contentfield on assistant messages is silently stripped during body rewriting. This causes the upstream to reject the request with:This PR preserves
reasoning_contentthrough the request forwarding pipeline so that multi-turn agent/tool-calling traffic works correctly with thinking-model endpoints.Root Cause
The ext_proc request body pipeline deserializes the inbound JSON into the OpenAI Go SDK struct (
openai.ChatCompletionNewParams), applies mutations (model name, reasoning mode, system prompt, memory), and re-serializes viajson.Marshal. The SDK (v1.12.0) does not definereasoning_contentonChatCompletionAssistantMessageParam— it's a DeepSeek/thinking-model extension not part of the official OpenAI spec. The unmarshal silently drops the field, and the marshal produces a body without it.The same request sent directly to the upstream (bypassing the router) works fine because the field is never round-tripped through a typed struct.
Fix
Uses the same
gjson/sjsonraw-JSON approach already established in the codebase forstream,model, andextra_bodymutations:messagesarray withgjsonand capture every assistant message'sreasoning_contentvalue (including empty strings — DeepSeek requires those too).sjson.This is zero-allocation for requests without
reasoning_content(the common case — the captured map is empty and the restore is a no-op).Files Changed
pkg/extproc/reasoning_passthrough.goextractReasoningContentFromMessages+restoreReasoningContentToMessagespkg/extproc/reasoning_passthrough_test.gopkg/extproc/request_context.goReasoningContentPassthroughfield toRequestContextpkg/extproc/processor_req_body_prepare.gopkg/extproc/processor_req_body_routing.goWhat is NOT changed
StreamingReasoningaccumulation and cache reconstruction already work correctly (covered by existingprocessor_res_cache_reasoning_test.go).reasoning_contentis a provider extension; even the latest SDK won't include it. The sjson approach is the right pattern here.reasoning_contentis present it's preserved; if absent, nothing changes.Test matrix:
✅ Multi-turn request with reasoning_content on assistant messages → preserved after SDK round-trip
✅ Empty string "reasoning_content":"" → preserved (DeepSeek requires this in thinking mode)
✅ Mixed message roles (system, user, assistant, tool) → only assistant messages affected
✅ Request with no reasoning_content → zero-cost no-op
✅ Existing streaming/cache reasoning tests continue to pass
Testing
go test -race -run TestReasoningContent ./pkg/extproc/...
Impact
Multi-turn agent and tool-calling traffic routed to DeepSeek V4 (and any other thinking-model endpoint that validates reasoning_content round-trips) will work correctly after this fix. Single-turn and non-thinking-model traffic is unaffected.
Part of #3547